Subject: Pacing proposal

From: [email-masked]
Date: Wed, 4 Aug 93 10:42:35 EDT

I haven't digested all the recent appends yet but let me give some
initial reactions to keep the discussion going.

Regarding Bernie's 7/23 append:
Shannon's 7/26 response captures exactly what I meant with RLWI.  The
receiver just gives back a solicited IPM with a new window size which
may be smaller, the same size, or larger than the last size granted.
The sender could compare the last and current size granted to see if
RLWI was honored or not by the receiver.  So I don't think ALWI is needed.

Regarding Robin's 7/23 append:
Thanks for the clarification.  If I understand you right, you are
simply describing the way 802.2 works (REJ could also be RR) and not
proposing something special that R1 needs to make happen.  If R1 RNRs
the first of a sequence of frames, the subsequent RR/REJ will have the
same N(R) as the RNR did, in normal operation.

Regarding Shannon's 7/26 append:
On including UI-frames in pacing, I see your point and am inclined
to agree.  Including UI-frames would provide more complete fairness
between circuits if there's lots of UI traffic.  Do other folks agree
with including UI-frames?  If anyone can describe briefly some products
that generate steady state LLC1 NetBIOS traffic, I would appreciate
getting a better sense of what the UI traffic patterns look like.

On exchanging initial pacing window size, I considered what you
suggested, Shannon, but I figured the benefit was not worth the
added complexity.  Since a TC is set up long before most of the
circuits on it, exchanged window sizes can't be used to reflect
any current buffer congestion conditions.  I can see a small benefit
giving "big" routers (in terms of static capacity), or high speed
links, a larger initial value than small routers/slow links,
but it takes some effort to go query these parameters and pick an
initial value.  Since the window size adapts to current conditions
anyway, I thought the "keep it simple" approach (fixed value of 1
or 2) might be best.

On requiring buffer reservation, the A#1 problem implementing APPN
ISR in reasonable-sized systems was the buffer pre-allocation
requirement.  With thousands of sessions the numbers quickly became
absurd, so the architecture was ok for workstations but not routers.
The architecture backed off and now allows statistical buffering.

The DLSw case is less severe since we are multiplexing SNA and NetBIOS
sessions onto a single circuit.  But I am still fairly sure that if
you analyze the number of circuits you'd like to support and the
memory requirements of full pre-allocation, at least the smaller
routers will find a full pre-allocation policy difficult to accept.
Consider that with a constant NWS of N, the worst case is you'd need
2N-1 buffers per circuit, and one buffer would have to be able to
hold the max-sized LAN frame you are supporting.  (Let me digress
for a second and say I think for source routing, DLSw routers
probably need to set LF bits to limit frame sizes to reflect their
buffering capability.  Allowing 16K frames from one t/r to another t/r
via DLSw would require a lot of buffer space - maybe 2K would be a
reasonable limit for a DLSw router.  I also think we must
propagate the LF bits end-to-end on the circuit.)  Over the next
few days, I think I will play with some combinations of memory sizes,
circuit, and pacing window numbers to make sure I can back this
argument up with more than the quick estimates I've done.

I agree that if you don't pre-allocate and fall short, your only option
is to not receive the data from the transport protocol, and this starts
to affect the other circuits.  So every effort should be made in a good
statistical buffer manager to avoid this condition.   And I think it's
the smaller implementations, not the larger, that have the greatest need
need for statistical buffer management.

On bagging the RESET/RESET ACK, it seems to me that most of the negatives
cited are based on assumptions of how it will be used (e.g., immediately
on receipt of an RNR - I guess my wording suggested this but I didn't
mean to require it).  I agree it would be better to take less drastic
measures first - withholding pacing, reducing NWS - so we don't reflect
hiccups onto the WAN.  I also agree that a RESET just cuts the problem
down statistically (I think better than half) and doesn't eliminate it.

Buffer managers I have seen all define different levels of pool usage
(e.g., ok, slowdown, constrained, critical), and I agree it's better that
an RNR not trigger an immediate RESET if the pool state is "ok",
for example.  One would use increasingly severe actions as things got
worse.  I'm concerned about the a priori assumption that if one does
a good design, one will never need a tool such as RESET.  Some scenarios
may cause severe congestion faster than withholding pacing can handle.
If we find this is true, it will be much harder to retrofit a RESET-like
mechanism in in the future than it is to keep it now but maybe give
everyone some guidelines that would hopefully prevent the misuse you
fear.  Of course what we do with this depends on the pre-allocation
policy, so I think it all needs some analysis as people think about
their individual implementations.

A couple of weeks ago, a number of people felt we needed a TC-level
flow control mechanism.  I would be interested to hear their current
thoughts - if we eliminate circuit RESET, we would probably eliminate
TC-level RESET also.

One last point on Shannon's append - Shannon, you liked unsolicited
pacing responses but RESET (with NWS=0) was the only one I had defined.
I messed up in my "new types" and meant to say the IPM types were
SOLICITED, RESET (not UNSOLICITED), and RESET ACKNOWLEDGEMENT (the
text of the proposal only referred to RESET).  If we did away with
RESET and RESET ACKNOWLEDGEMENT, what would the rules be?  Would you
reinstate UNSOLICITED and allow only non-zero NWSs?

Regarding Anand's 7/30 append:
In the RESET scheme, the sender who has been reset does not request
the receiver to open the window.  After receiving the RESET ACK, the
receiver owes the sender a SOLICITED IPM at some future point.  So
there is no timer.  I was expecting that a RESET would be the only
way the receiver would send NWS=0, so there was no mechanism defined
for a SOLICITED IPM with NWS=0.

NWS can be used to increase or decrease the window to arbitrary values,
not just increment by 1.  I would expect that a receiver in trouble
would start halving the window - decrementing is too slow.  As I just
told Shannon, I did not define an unsolicited non-zero NWS mechanism,
but we could add that to the scheme if people want (no ACK required).

On Urgent data, I know it's not going to bypass data in the network,
but I was guessing that faster delivery to the application would be
important to prevent some bad situations.  I haven't had time to
really explore scenarios, though.  I would love it if someone could
find an hour to see if they could create any deadlock scenarios - it
may be that since DLSw does not have concatenation of circuits we cannot
have the circular wait conditions that more general architectures must
deal with.

There's one other TCP wrinkle to consider, and that's poor performance
at small pacing window sizes, since TCP runs timers trying to coalesce
data before sending it out.  DLSw may need to do a "push" function in
some situations - if anyone has some experience with real-world TCP
window sizes and can give insight here, it would be appreciated.

Peter

---

From: "Shannon Nix" <[email-masked]>
Date: Wed, 11 Aug 93 08:25:44 EST

On Wed, 4 Aug 93 10:42:35 EDT, <[email-masked]> wrote:

...
>anyway, I thought the "keep it simple" approach (fixed value of 1

  It is not meant to reflect current buffer conditions, but rather an 
  implementor's choice for the initial value ( this might reflect system
  capacity and/or a preconfigured max circuits??).  The DLSw sig already 
  has a  group working on capabilities exchange.  I would think this sort 
  of feature is suited to that mechanism. 

  Further, who is to say that capabilities exchange is limited to TC setup?
  Using something like non-activation XIDs, couldn't one react to current   
  buffer  and circuit loads?

>argument up with more than the quick estimates I've done.

  Let me explain what I mean by preallocation:

     For each circuit, an optimal window size should be maintained which 
     I'll call:       NWS(o).

  I am NOT proposing that that circuit allocate that some multiple of NWS(o)
  window ( m where 1 < m < 2) and hang on to it until W(o) changes.

  What I DO propose is that:

    1) Prior to sending an IPR the receiver attempt to allocate NWS(o) 
       buffers.  

    2) If however, current buffer availability does not permit that, then 
       the receiver "falls back" to a temporary window size which is 
       based on the current number of buffers that circuit can fairly
       allocate at that instant ( this could be zero).  I'll call this 
       temporary fall-back value NWS(t). 

    3) When a frame has been reliably dispatched by the receiver, the
       buffer(s) return to the system pool of available buffers.

   W(o) should be computed from the following factors:

       1) System Buffer Capacity
       2) Current Circuit Load
       3) Receipt of RLWIs ( is the current window being utilized?) 
       4) Priority of the circuit
       5) Number of pending ( undispatched) frames for the circuit
          ( is my boundary media or target end station unable to accept the 
            current rate of frame arrival?)

   W(t) should be computed on the following factors:

       1) Current buffer availability at that instance
       2) Current Circuit Load
       3) Priority of the circuit

   NOTE:
        The aggregate system NWS(o) may still statistically overcommit
        the system buffer capacity. Granted, this is not to the degree that 
        a buffer-on-demand policy could:  BUT... one doesn't have 
        to be as statistically "safe" as a buffer-on-demand policy 
        because if one receives a "run on the buffer bank" they simply 
        fall back using NWS(t)s until the situation goes away or 
        the system tunes itself.  With buffer-on-demand, when one is 
        "found short" the whole TC gets gummed up.

   FURTHER:
        The degree to which the aggregate NWS(o) overcommits the system
        buffer capacity could (should?) be tuned relative to the instances
        of "fall-backs".  This would cause the system to "tune itself" 
        to the burstiness of the traffic over time.

>need for statistical buffer management.

  By larger, I was refering to number of concurrent sessions supported.

>their individual implementations.

It certainly has increased utility in a system using a "buffer-on-demand"
policy.

>reinstate UNSOLICITED and allow only non-zero NWSs?

If a circuit sends a solicited  IPR with an NWS of zero ( which I believe 
he should be allowed to do), then that circuit must "reopen the window"
with an UNSOLICITED IPR carrying a non-zero NWS.  That seems the only
requirement for UNSOLICITED IPRs.  However, there are other useful(??) areas
an implementor could employ them.

>have the circular wait conditions that more general architectures must

  I think by eliminating piggy-backed PRs you elimated the dead-lock.  I 
  believe the issue for urgent IPRs is solely one of perfomance.

>window sizes and can give insight here, it would be appreciated.

Certainly IPRs need to find their way out the door rather quickly. I think 
we should try to specify the required TC characteristics independent of 
TCP though.  I would be nice, of course, if those characteristics
were easily implementable using TCP!

>Peter

Rgds, Shannon
(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)
(*)(*)(*)(*)(*)(*)(*)(*)   Shannon Nix    (*)(*)(*)(*)(*)(*)(*)(*)(*)
(*)(*)(*)(*)(*)(*)(*)      Netlink, Inc.     (*)(*)(*)(*)(*)(*)(*)(*)
(*)(*)(*)(*)(*)(*)(*)  3214 Spring Forest Rd.   (*)(*)(*)(*)(*)(*)(*)
(*)(*)(*)(*)(*)(*)(*)  Raleigh N.C. 27604 USA   (*)(*)(*)(*)(*)(*)(*) 
(*)(*)(*)(*)(*)(*)(*)  Phone:  (919)-878-8612   (*)(*)(*)(*)(*)(*)(*) 
(*)(*)(*)(*)(*)(*)(*)  Fax:    (919)-872-2132   (*)(*)(*)(*)(*)(*)(*) 
(*)(*)(*)(*)(*)(*)(*) Internet: [email-masked] (*)(*)(*)(*)(*)(*)(*) 
(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)